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Background of the Inv ntion 

1 . Field of the Invention 

[0007] The present invention relates to apparatus for processing image 
data. 

2. Description of the Related Art 

[0008] Systems for processing image data, having a processing unit, 
storage devices, a display device and a stylus-like manually operable input 
device (such as a stylus and touchtablet combination) are shown in United 
States Patents 5,892,506; 5,786,824 and 6,269,180 all assigned to the 
present Assignee. In these aforesaid systems, it is possible to perform many 
functions upon stored image data in response to an operator manually 
selecting a function from a function menu. 

[0009] Recently, in such systems as "FIRE" and "INFERNO", licensed by 
the present Assignee, the number of functions that may be performed have 
increased significantly. In addition, for example, there has been a tendency 
towards providing functions for special effects, compositing and editing on the 
same platform. 

[0010] Function selection is often done via graphical user interfaces in 
which menus are displayed from which a selection may be made. A function 
selection using a menu is achieved by moving a cursor over to a selection 



position within the menu by operation of the stylus. The particular function 
concerned is selected by placing the stylus into pressure; an operation 
logically similar to a mouse click. Menus of this type are used in systems 
where stylus-like input devices are preferred, in preference to pulldown 
5 menus. 

[0011] In addition to there being a trend towards increasing the level of 
functionality provided by digital image processing systems, there has also 
been a trend towards manipulating images of higher definition. Initially, many 

10 systems of this type were designed to manipulate standard broadcast 
television images such as NTSC or PAL. With images of this type, it is 
possible to display individual frames on a high definition monitor such that the 
displayed images take up a relatively small area of the monitor thereby 
leaving other areas of the monitor for displaying menus etc. Increasingly, 

15 digital techniques are being used on high definition video images or images 
scanned from cinematographic film. Such images have a significantly higher 
pixel definition. Consequently, even when relatively large monitors are used, 
there may be very little additional area for the display of menus. 

20 [0012] Furthermore, operators and artists are under increasing pressure to 
speed up the rate at which work is finished. Being able to work with systems 
of this type quickly and efficiently is not facilitated if complex menu structures 
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are provided or manipulation tools are included that are not intuitive to the 
way artists work. 



Brief Summary of the Invention 
5 [0013] According to a first aspect of the present invention, there is provided 
apparatus for processing image data, comprising processing means, storage 
means, display means and stylus-like manually operable input means, 
wherein said processing means is configured to perform functions upon 
image data in response to an operator manually selecting a function from a 

10 function menu; said processing means responds to a first user-generated 
input command so as to display a plurality of function gates at a cursor 
position; movement of the stylus-like manually operable input means so as to 
move said cursor through one of said function gates results in a related menu 
being displayed; and manual selection of a function from said displayed 

15 menu results in the selected function being performed upon said image data. 

Brief Description of the Several Views of the Drawings 
[0014] Figure 1 shows a system for processing image data; 
[0015] Figure 2 shows details the computer system shown in Figure 1\ 
20 [001 6] Figure 3 shows illustrates the display of the prior art; 

[001 7] Figure 4 shows the display of Figure 3 with graphically displayed 
menus as is known in the prior art; 
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[0018] Figure 5 shows an example of a scene graph defining how a 
complex scene is rendered; 

[0019] Figure 6 is the monitor of Figure 1 displaying a high definition image; 
[0020] Figure 7 shows a portion of the image shown in Figure 6 with user 
5 interface gates; 

[0021] Figure 8 shows an abstracted view of the gates shown in Figure 7; 
[0022] Figure 9 shows the high definition image of Figure 6 with an overlaid 
upper menu; 

[0023] Figure 10 shows the high definition image of Figure 6 with a lower 
10 menu; 

[0024] Figure 11 shows the high definition of Figure 6 with a menu to the 
left; 

[0025] Figure 12 shows the high definition image of Figure 6 with a menu to 
the right; 

15 [0026] Figure 13 identifies operations performed by the processing unit 
shown in Figure 2; 

[0027] Figure 14 details procedures identified in Figure 13] 
[0028] Figure 15 details procedures identified in Figure 14] 
[0029] Figure 16 details procedures identified in Figure 15] 
20 [0030] Figure 1 7 identifies a first alternative embodiment of the present 
invention; 

[0031] Figure 18 identifies further alternative embodiments of the present 
invention. 
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[0032] An embodiment of the invention will now be described by way of 
example only with reference to the above drawings. 

5 Detailed Description 
Figure 1 

[0033] Apparatus for processing image data is illustrated in Figure 7. In this 
example a computer system 101 supplies output signals to a visual display 
unit 102. The visual display unit 102 displays images, menus and a cursor, 

10 and movement of said cursor is controlled in response to manual operation of 
a stylus 103 upon a touch table 104. In addition, input data is also supplied to 
the computer system 101 via a keyboard 105. Keyboard 105 is of a standard 
alpha numeric layout and includes a spacebar 106. Manual operation of the 
spacebar 106 provides a first input command in a preferred embodiment 

15 resulting in a selection device being displayed at the cursor position. The 

selection device identifies a plurality of function types (for example four) each 
having an associated displayable menu. In response to a second input 
command, preferably received from the stylus 103, the cursor is moved over 
one of the identified function types. Thereafter, having moved the cursor over 

20 a displayed type, the aforesaid menu associated with the function type over 
which the cursor has been moved is displayed. In this way, a user is given 
rapid access to a menu of interest without said menu being continually 
displayed over the working area of the VDU 102. 
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Figure 2 

[0034] Computer system 101 is illustrated in Figure 2. System bus 201 
provides communication between a central processing unit 202, random 
5 access storage devices 203, a video card 204, disk storage 205, CD ROM 
reader 206, a network card 207, a tablet interface card 208 and a keyboard 
interface card 209. Typically, the central processing unit may be an Intel 
based processor operating under the Windows operating system. Program 
instructions for the central processing unit 202 are read from the random 
10 access memory device at 203. Program instructions are preferably received 
via a CD ROM 210 (or similar computer-readable medium) for installation 
within the storage system of disk drive 205 via the CD ROM reader 206. 

[0035] Network card 204 supplies output signals to monitor 102 with input 
15 signals from the tablet 104 being received via a tablet interface 208 and input 
signals from keyboard 105 being received via the keyboard interface 209. 
Network interface 207 allows the system to exchange files with a server or 
other networked stations. 

20 Figure 3 

[0036] A monitor 301 , of a prior art system and not that shown in Figure 1, 
is illustrated in Figure 3. The monitor is displaying a video image 302 
consisting of a plurality of frames played over a period of time at standard 
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broadcast definition. The monitor has a substantially higher definition, thereby 
ensuring that there is plenty of space around the image 302 for graphical 
interfaces to be displayed. The skilled reader will understand that it is the 
entire system that is prior art and not specifically the high-definition monitor. A 
5 similar monitor could be used in an embodiment of the present invention. 

Figure 4 

[0037] Monitor 301 is shown in Figure 4 with a plurality of menus, such as 
menu 304 and menu 305, displayed around video image 302. In this way, 

10 many control functions may be selected by appropriate operation of a stylus 
upon a touch-tablet. A function of interest is selected by placing the cursor 
over a soft button. The button is then depressed by placing the stylus into 
pressure. This may result in a function being performed upon the image 
directly or, alternatively, may result in an appropriate sub-menu being 

15 displayed so that appropriate control may be made in response to user input. 

[0038] It can be appreciated that the working space displayed on monitor 
301 has become somewhat complex if all available functions are to be 
displayed. 

20 

Figure 5 

[0039] The number of possible functions available to an artist has 
increased and there is a trend for more and more of these functions to be 
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used concurrently to produce a particular effect. Furthermore, it is preferable 
for the nature of the functions to be stored as definitions or metadata 
whereafter their implementation takes place in real-time. Thus, the process of 
compositing etc requires many functions to be performed as part of a final 
5 rendering operation rather than partially processed work being stored and 

then processed again. Consequently, many functions may be required and in 
order to make modifications an artist is required to identify a particular 
function of interest. 

10 [0040] In order to provide artists with a representation of the nature of a 
function being performed, the structure of the processing operations may be 
displayed as a process tree, as illustrated in Figure 5. Process trees generally 
consist of sequentially linked processing nodes, each of which specifies a 
particular processing task required in order to eventually achieve an output in 

15 the form of a composited frame or video sequence. Traditionally, an output 

sequence 501 will comprise both image data and audio data. Accordingly, the 
composited scene will require the output from an image keying node 502 and 
the output from a sound mixing node 503. In this example, the image keying 
node 502 calls on a plurality of further processing nodes to obtain all the input 

20 data it requires to generate the desired image data, or sequence of 

composited frames. In the example, the desired output image data includes a 
plurality of frames within which a three-dimensional computer generated 
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object is composited, as well as a background also consisting of a plurality of 
three-dimensional objects superimposed over a background texture. 

[0041] The image keying node 502 requires a sequence of frames 
originating from node 504. Each frame undergoes a colour correction process 
at node 505 followed by a motion tracking process at a motion tracking 
process node 506. Modelled 3D objects are generated by a three- 
dimensional modelling node 507 and a texture is applied to these objects by 
a texturing node 508. After being textured, lighting is applied by an artificial 
light processing node 509, followed by a scaling operation performed by a 
scaling node 510. Tracking node 506 is then responsible for combining the 
computer generated object with the image frames. To generate the 
background, image processing node 502 also requires a uniform texture from 
a texturing node 51 1. Colour correction is applied to this texture by means of 
colour correction node 512 a further three-dimensional modelling node 513 
generates further objects upon which lighting is applied by node 514 followed 
by tracking performed by node 515. Consequently, image keying node 502 
may now composite the foreground objects with the background. 

[0042] Each node illustrated in Figure 5 will have an associated menu of 
controls allowing modifications to be made at that particular point in the 
overall image processing exercise. Thus, when modifications are made at the 
menu level, it is necessary for a database to be established so as to oversee 
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the relationship between manual input commands being made and their 
associated node at which the modifications are to take effect. Thus, the 
complexity of images results in a greater requirement for the display of 
control menus so as to allow full control to be given to an artist during a 
5 compositing exercise. It will be appreciated that other methods of storing data 
associated with processing operations exist, and that the invention is not 
limited to image processing apparatus which operates in the way described 
herein. 

10 Figure 6 

[0043] Problems associated with the availability of free monitor space are 
made worse when the definition of images being processed is increased. 
Figure 3 shows a prior art example of a standard television broadcast image 
being processed. However, as illustrated in Figure 6, the present invention is 

15 particularly directed towards the processing of higher definition images such 
as images derived from cinematographic film. Thus, a high definition image 
has been loaded of a definition such that, when displayed, as illustrated in 
Figure 6, the whole of the available display space of visual display unit 102 is 
used for displaying the image frames. Even with very large visual display 

20 units, it is recognised that artists must work with material at an appropriate 
definition so as to ensure that the introduction of visible artefacts is 
minimised. However, a problem with displaying images at this definition, as 
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illustrated in Figure 6, is that the monitor does not provide additional space 
for the display of menus alongside the displayed high definition images. 

[0044] Region 602 of the high definition image 601 is shown enlarged in 
5 Figure 7. A cursor 603 is shown in Figure 6 at a selected position. After being 
placed in this selected position, an artist operates spacebar 106 of the 
keyboard 105 resulting in a selection device being displayed at the cursor 
603 position. Clearly other ways of activating the selection device may be 
used apart from the space bar, for example other keys on the keyboard, a 
10 button on the stylus, and so on. 

Figure 7 

[0045] A displayed selection device providing four selection regions, that 
have been identified as "gates", is shown at 701 in Figure 7. Each gate of the 

15 displayed device 701 identifies a function type and each of said function 

types has an associated displayable menu. After activating the spacebar, the 
selection device 701 is located around the position of the displayed cursor 
603. The selection device 701 remains displayed after the space bar has 
been activated. A further activation of the space bar removes the device 701 . 

20 In addition, device 701 is also removed if the stylus is activated so as to move 
the cursor 603 through one of the gates 702 to 705. Moving the stylus 103 in 
an upwards direction results in the displayed cursor 603 passing through the 
"viewer" gate 702. In response to passing the cursor 603 through the viewer 
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gate 702, a viewer menu is displayed in an upper portion of the screen. 
Similarly, by moving the stylus 103 in a downward direction, the cursor 603 is 
passed through a tool control gate 703 (a transform in this example), 
identified as a transform tool in Figure 7. By moving the stylus 103 to the left, 
the cursor 603 passes through a "layer" gate 704 resulting in an associated 
menu being displayed to the left of the image. Furthermore, by moving the 
stylus 103 to the right, the displayed cursor 603 is taken through the tools 
gate 705, resulting in an appropriate menu being displayed to the right of the 
image. 

[0046] The particular function types available are relevant to the application 
being performed in the preferred embodiment. However, it should be 
appreciated that similar techniques may be used in different environments. 
Within the same application, it is possible that different views may be called 
and one or more of said views may have an interface device relevant to that 
particular view. For example, a schematic view may be shown or a player 
view may be shown. Upon calling the interface device (by activation of the 
space bar) the interface device may be relevant to schematic operations 
when the schematic view is shown and may be relevant to player operations 
when the player view is shown. The schematic viewer displays the entire 
composition (that is to say the whole graph). The user usually has a node 
selected in the graph. When the user displays the schematic gate device it 
will preferably display the schematic starting from the current selection. This 
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will show the user everything in the scene that generated the current 
selection and is therefore a filtered version of the schematic view. 

Figure 8 

[0047] An abstracted interface is illustrated in Figure 8. In response to a 
first input command, an interface device 801 is displayed at a cursor 806 
position. In this embodiment, this first input command consists of the 
spacebar of a keyboard being depressed. The interface device identifies a 
plurality of function types (802, 803, 804, 805) and by passing a cursor 806 
through one of these function types, an appropriate menu is displayed 
Although the menu can be displayed in any part of the screen, it is preferably 
displayed at a location related to the gate through which the cursor has 
passed. Thus, if the cursor 806 moves to the left, preferably a left menu is 
displayed; if the cursor 806 moves to the right, preferably a right menu is 
displayed; if the cursor 806 moves upwards, preferably an upper menu is 
displayed; and if the cursor 806 moves downwards, preferably a lower menu 
is displayed. 

Figure 9 

[0048] In a preferred embodiment, movement of cursor 602 in response to 
stylus 103 in an upwards direction through gate 702 results in a movement of 
viewer gate menu 901 being displayed in an upper portion of the screen. The 
viewer gate menu is used to set viewer specific options such as render pre- 
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sets for three-dimensional players or filtering for schematics. The viewer 
menu relates directly to the viewer in focus and the name of the viewer in 
focus preferably appears in the gate user interface. The displayed menu 
takes up the same width as a tool panel user interface and it is locked to the 
5 top of the user interface regardless of how many viewers are present. The 
panel is fully opaque and sits over all other panels. Upon leaving the viewer 
menu the menu itself disappears thereby returning the full screen to the 
image under consideration. 

10 Figure 10 

[0049] Moving the cursor 602 in a downward direction, through gate 703, 
results in a current tool menu 1001 (a transform in this example) being 
displayed in a lower region of the screen of monitor 102. The current tool 
menu is used to interact with the current tool. Gate 703 is only available if one 

15 tool has been selected. Thus, the gate relates directly to the current tool 

under consideration. The name of the current tool preferably appears in the 
gate user interface. The menu is locked to the bottom of the player in focus 
and use is also made of the transport tool user interface. 

20 [0050] After use has been made of the current tool menu, the menu is 

removed by activating spacebar 106 again, thereby making the whole screen 
available for the whole image. Activation of an "escape" has a similar effect. 
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Figure 11 

[0051] Upon moving cursor 602 in a leftward direction through gate 704, a 
layer gate menu 1 101 is displayed. The layer menu is used to select layers 
and the layer user interface takes up the same width as a layer list. It is 
locked to the left side of the user interface regardless of how many viewers 
are present. The panel is fully opaque and sits over all other panels. The 
layer gate menu 1 101 only contains details of the layers; the layer list is not 
expandable and there is no value column. A user can set whether a layer is 
visible or not visible and the layer menu 1101 disappears after the cursor 
exits to a new area. 

Figure 12 

[0052] Upon moving cursor 602 in a rightwards direction through gate 705 
tools menu 1201 is displayed. The tools menu is used to select the current 
tool and is only available when only one layer has been selected. The tools 
gate menu takes up the same width as the layer list and is locked to the right 
side of the interface regardless of how many viewers are present. The panel 
is fully opaque and sits over all other panels. The tools menu 1201 contains a 
filtered version of the schematic showing only the tools associated with a 
selected object. The menu disappears after the cursor has been moved out 
of the menu area. It should be appreciated that these particular menu 
selections are purely an application of the preferred embodiment and many 
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alternative configurations could be adopted while invoking the inventive 
concept. 

Figure 13 

[0053] Operations performed by the processing unit 202 in order to provide 
the functionality described with reference to Figures 6 to 12 is identified in 
Figure 13. After power-up an operating system is loaded at step 1301 
whereafter at step 1302 the system responds to instructions from a user to 
run the compositing application. 

[0054] At step 1303 data files are loaded and at step 1304 the application 
operates in response to commands received from a user. At step 1305 newly 
created data is stored and at step 1306 a question is asked as to whether 
another job is to be processed. When answered in the affirmative, control is 
returned to step 1303 allowing new data files to be loaded. Alternatively, if the 
question asked at step 1306 is answered in the negative, the system is shut 
down. 

Figure 14 

[0055] Procedures 1 304 relevant to the present preferred embodiment are 
illustrated in Figure 14. At step 1401 a keyboard operation is captured and at 
step 1402 a question is asked as to whether the spacebar has been 
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activated. If answered in the negative, control is returned to step 1401 else 
control is directed to step 1403. 

[0056] In response to the spacebar being activated and detected at step 
1402, selection gates 701 are displayed at step 1403. At step 1404 a 
question is asked as to whether the spacebar has been released and if 
answered in the affirmative, the selection gates are removed. Alternatively, if 
the question asked at step 1401 is answered in the negative, control is 
directed to step 1406 such that the application responds to further cursor 
movement. 

Figure 15 

[0057] Procedure 1406 is detailed in Figure 15. At step 1501 cursor 
movement is captured and at step 1502 a question is asked as to whether 
the cursor has moved across the upper gate 702. If answered in the negative, 
control is directed to step 1505, but if answered in the affirmative the upper 
menu (the viewer menu in the preferred embodiment) is displayed at step 
1503 and the system responds to menu selections made at step 1504. 

[0058] At step 1504 a question is asked as to whether the cursor has 
crossed the lower gate 703 and if answered in the negative control is directed 
to step 1508. If answered in the affirmative, to the effect that the cursor did 
cross the lower gate 703, the lower gate menu (selected tool menu in the 



18 



preferred embodiment) is displayed at step 1506 and responses to selections 
are made at step 1507. 

[0059] At step 1508 a question is asked as to whether the cursor has 
5 crossed the left gate 704 and if answered in the negative control is directed to 
step 705. In answered in the affirmative, the left gate menu (the layer menu in 
the preferred embodiment) is displayed at step 1509 and responses to 
selections are made at step 1510. 

10 [0060] At step 151 1 a question is asked as to whether a cursor has crossed 
the right gate 705. If answered in the affirmative, the right gate menu (the 
tools menu in the preferred embodiment) is displayed at step 1512 and the 
system responds to manual selections at step 1513. 

15 Figure 16 

[0061] Procedures 1504 for responding to input selections are detailed in 
Figure 16. At step 1601 a position is captured when the stylus 103 is placed 
in pressure. 

20 [0062] At step 1602 a question is asked as to whether a menu has been 
closed, either as a result of a "close menu" button being operated or, for 
certain menus, whether the stylus has been taken outside the menu area. If 
answered in the affirmative, the menu is closed at step 1603. 
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[0063] If the question asked at step 1602 is answered in the negative, a 
question is asked at step 1604 as to whether a function has been selected. If 
answered in the affirmative, the function is called at step 1605. 

[0064] Procedures 1507, 1510 and 1513 are substantially similar to 
procedures 1504 shown in Figure 16. 

Figure 17 

[0065] An alternative embodiment is illustrated in Figure 17. Instead of the 
substantially circular device being divided into four sections, allowing four 
function menus to be selected, a circular device 1701 is divided into three 
sections from which three function devices may be selected. 

Figure 18 

[0066] A further alternative embodiment is illustrated in Figure f S in which a 
substantially circular device 1801 has been divided into six sections allowing 
six functional menus to be selected. In the preferred embodiments disclosed 
herein, the selection device has a substantially circular shape. It should also 
be appreciated that other shapes, such as quadrilaterals etc may be adopted 
as an alternative. 
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